Method for automatic alignment of tilt series in an electron microscope

ABSTRACT

It may be desirable to obtain three-dimensional information on a sample  2  to be studied in an electron microscope. Such information can be derived from a tilt series  2 - i  of the sample and a subsequent reconstruction of the three-dimensional structure by means of a computer algorithm. For a proper reconstruction of the structure in the volume of the sample it is important that the measurement geometry be known; therefore it is important that the images be properly aligned. Therefore markers  8 - i  (e.g. gold particles) are applied to the sample, which markers yield straight lines  10 - i  as the sample is rotated and projections of that rotated sample are made onto one image plane. According to the invention the straight lines are recognized, which gives the possibility to identify the individual markers in the images of the tilt series, and to align those images on the basis of the information thus obtained.

The invention relates to a method for automatic alignment of tilt seriesin an electron microscope, comprising:

-   -   applying markers to a sample to be imaged by the electron        microscope;    -   providing a tilt series of images of the sample;    -   identifying a first set of candidate markers in each of the        images in the tilt series, and    -   attributing at least one probability parameter to each candidate        marker in each image.

In electron microscopy it may be desirable to obtain three-dimensionalinformation on a sample to be studied in the microscope. Suchinformation can be derived from a tilt series of the sample and asubsequent reconstruction of the three-dimensional structure by means ofa computer algorithm. In a Transmission Electron Microscope (TEM) thesample may have a thickness of a few hundred nanometers, the resolutionof the microscope being a few nanometers or even less. A tilt series isa series of images of one sample in which individual images are recordedwhile the sample is irradiated at various angles of incidence of theelectron beam exposing the sample, e.g. a series of 141 images in a tiltinterval from −70° to +70° in steps of 1°. For a proper reconstructionof the structure in the volume of the sample, in particular at highresolutions, it is important that the measurement geometry be known;therefore it is important that the images be properly aligned. Lack ofalignment of the individual pictures may occur, for example, due totemperature drift of the microscope during data acquisition, sampleshrinkage or due to mechanical imperfections of the sample stage and thetilt mechanism, as a result of which the stage position cannot bedetermined with nanometer accuracy.

In a known method for alignment of images in tilt series, markerparticles, e.g. gold particles having a size of typically a few tens ofnanometers, are applied to the sample. Each marker particle appears ineach image of the tilt series, thus offering a position reference foreach image. Because of the great number of images in a tilt series thesemarkers should preferably be recognized and identified automatically.Algorithms for automatically recognizing markers in an image have theability to indicate a structure in an image as a marker withoutproviding surety that such structure is indeed a marker. For this reasonthe indicated structure is labelled as a candidate marker, with someprobability that such labelled structure is indeed a marker.

A method for identifying a set of candidate markers in an image and forattributing at least one probability parameter to each candidate markerin the image is known from an article entitled “Scale-Space Signaturesfor the Detection of Clustered Microcalcifications in DigitalMammograms”, IEEE Transactions on Medical Imaging, Vol. 18, No. 9,September 1999, pp.774-786. This article describes the way in whichcandidate markers are found in an image: see, for example, section III,subsection C of the article. Also, how to attribute a reliabilityparameter may be derived from this article: see, for example, sectionIIIC, FIG. 9 and the corresponding description on page 779 left column,first full section, in which the size and the contrast are used todecide whether or not an indicated structure is recognized. No furtherindications for recognizing markers can be derived from this article.

It is an object of the invention to provide a method for automaticallyaligning tilt series in an electron microscope by recognizing markers inthe sample to be studied. According to the invention this object may besatisfied in that the method further comprises:

-   -   selecting a second set as a subset of candidate markers from the        first set of candidate markers on the basis of said at least one        probability parameter;    -   projecting the candidate markers in the second set onto a sole        image;    -   applying a fitting algorithm to determine a set of parallel        straight lines or very elongate ellipses best fitting the        candidate markers in the sole image;    -   aligning the images in the tilt series on the basis of the        identified candidate markers.

In the method of finding the markers in the images of the tilt seriesone may start by identifying candidate markers in each of the images inthe tilt series in a rough manner. When applying this rough selectionmethod, many candidate markers that are not real markers (false positivecandidates) will be part of its selection result. By applying theselection step on the basis of the probability parameter according tothe invention, a lot of false positive candidates can be deleted, thusproviding a higher amount of reliability in the second set of candidatemarkers, the second set being a subset of the first set of candidatemarkers. By projecting the candidate markers in the second set onto asole image, the real markers will be situated on a set of parallelstraight lines, the direction of these straight lines beingperpendicular to the rotation axis used as the tilt axis in obtainingthe tilt series of images of the sample. The other candidate markers(i.e. the remaining false positive ones) will be scattered in a more orless arbitrary way over the sole image obtained by the projection step.By applying a proper fitting algorithm the parallel straight lines orvery elongate ellipses fitting the candidate markers in the sole imagecan be determined, preferably by using a computer. Sometimes it mayhappen that very elongate ellipses, instead of real straight lines, willoccur in the sole image. This artefact is caused by the tilt axis notbeing exactly perpendicular to the optical axis of the electron beam.After having determined the set of straight lines, the direction of thetilt axis is known and the true markers (i.e. the ones that are situatedon the set of straight lines) can be identified as such and subsequentlyused for aligning the images in the tilt series. This alignment may thenbe carried out in a manner known per se.

In a preferred embodiment of the invention the fitting algorithm used todetermine the set of parallel straight lines comprises the Houghtransformation. In particular in the use of a computerised recognitionprocess for recognizing straight lines in the multitude of candidatemarker points, the Hough transformation (known per se) has proven to bea reliable and stable algorithm for this recognition process.

In another embodiment of the invention the fitting algorithm used todetermine the set of parallel straight lines or to determine a set ofvery elongate ellipses is constituted by the Generalized Houghtransformation. It may happen that candidate markers belonging to onestraight line are scattered in such a way that they are not alignedaccording to a straight line but—due to some undesirable influences—theyare approximately arranged according to very elongated ellipses. In suchcircumstances the Generalized Hough transformation (known per se) is agood algorithm to find the directions of the long axes of the ellipsesbest fitting the candidate markers so arranged.

In still another embodiment of the invention a cross correlation processis applied to the images of the tilt series before identifying candidatemarkers in each of the images in the tilt series. One may start with arough mutual alignment of the individual images of the tilt series,which has the effect that the projections of the markers onto the soleimage are approximately situated on straight lines rather than beingscattered more or less at random over the sole image; this rough firstalignment step provides a concentration of the markers on the straightlines, and has the advantage that the set of straight lines found by thefitting algorithm will contain very few false positive ones. This roughfirst alignment step may be constituted by said cross correlationprocess.

In still another embodiment of the invention the probability parameteris derived from at least one of the quantities: size of the marker andlocal contrast of the marker. These quantities can be derived relativelyeasily by means of a computerized analysis of the images, and they haveproved to be reliable quantities for determining—in a first selectionprocess—whether an initially indicated structure might be identified asa candidate marker.

In an embodiment of the invention comprising the Hough transformation asthe fitting algorithm used to determine the set of parallel straightlines, the fitting algorithm further comprises:

-   -   deriving for each candidate marker a sine-shaped curve based on        the coordinates of the corresponding candidate marker, according        to the Hough transformation;    -   deriving from the sine-shaped curves a number of histograms        indicating, for each direction, the relation between the density        of candidate markers and the line distance parameter according        to the Hough transformation;    -   applying an entropy operation to each of the histograms,        resulting in a set of entropy parameters, one entropy parameter        for each histogram;    -   establishing the minimum value in the set of entropy parameters;    -   identifying the histogram corresponding to said minimum value as        the one showing the highest degree of peak diversity;    -   selecting from the latter histogram a number of peaks;    -   deriving from each peak position in the histogram the        corresponding line distance parameter according to the Hough        transformation.

The above set of technical measures describes in more detail the way inwhich the parameters identifying the set of straight lines containingthe true marker positions are determined. Such process can be executedin a convenient way by using a computer.

The invention will be described in more detail hereinafter withreference to the Figures, in which identical reference numerals denotecorresponding elements, and wherein:

FIG. 1 a: is a schematic view of a sample being exposed during a tiltseries;

FIG. 1 b: is a schematic image in which the markers of a tilt series areprojected;

FIG. 2 a: is an image of an object in which all candidate marker pointsare indicated;

FIG. 2 b: is an image of an object in which a rough selection of markerpoints has been made;

FIG. 2 c: represents the projection of the selected candidate markers ofthe tilt series;

FIG. 3 a: is a graphical representation of a straight line for thepurpose of explaining the application of the Hough transformationaccording to the invention;

FIG. 3 b: is a is graphical representation of the Hough transformationof a bundle of straight lines for the purpose of explaining theapplication of the Hough transformation according to the invention;

FIG. 4: is a histogram representing the number of markers as seen in afirst direction;

FIG. 5 a: is an auxiliary figure for the purpose of explaining theselection of the line direction corresponding to the tilt axis;

FIG. 5 b: is an auxiliary figure representing the peakiness of thehistograms according to FIG. 4, for selecting the line directioncorresponding to the tilt axis;

FIG. 6: is a histogram corresponding to the line direction of the tiltaxis.

FIG. 1 shows a schematic view of a sample 2 being exposed by an electronbeam 4 during a tilt series. The sample may have a thickness typicallyof the order of magnitude of a few hundred nanometers. During a tiltseries the sample is rotated about a tilt axis 6, which in this Figureis perpendicular to the plane of drawing. The rotation takes placethrough some fixed angle, e.g. from −70° to +70° with steps of 1°, whichmeans that, in such a tilt series, 141 exposures are made. The variousorientations of the sample during the tilt series are indicated as 2-1,2-2, 2-3 etc. The sample is provided in a known way with markerparticles (markers) in the form of gold particles having a size oftypically a few tens of nanometers. This can be done by immersing thesample in a fluid provided with a gold solution, which fluids arecommercially available. After immersion of the sample in the fluid, anumber of gold particles remain on the sample in such a way that theycan serve as markers. These markers are indicated in the figure byreference numerals 8-1, 8-2 etc. It should be remarked that, in general,the direction of the tilt axis 6 with respect to the image of the sampleis not known, e.g. because of the image rotation caused by the use ofmagnetic electron lenses. During rotation of the sample in a tiltseries, the alignment of the sample is not known exactly, for variousreasons such as temperature drift of the sample stage during the series,or play in the moving parts of the sample stage. In order to obtain agood spatial reconstruction of the image of the sample, alignment of theimages of the tilt series must be possible, for which purpose themarkers serve as reference points for the alignment. Therefore, anyparticular marker in a given image must be recognized in the otherimages. The invention offers an easy and reliable method of recognizingthe markers in the individual images.

FIG. 1 b illustrates the principle as to how the markers contribute toprojected lines, such that individual markers may be recognized. Thefigure provides a view of the sample 2 as seen from the direction of theincident electron beam. The rotation axis 6 is located in the plane ofthe drawing, as well as the image plane 2 on which all markers areprojected. Some markers 8-1, 8-2, 8-3, 8-4 etc. are shown in the figure.In executing a tilt series the markers 8-i are turned about the axis,thereby describing part of a circle having a plane that is perpendicularto the axis of rotation 6. The orbits of the markers along the circleare projected onto the image plane 2, which is illustrated withreference to marker 8-3. Each image i of the tilt series provides aprojection 12-i on image plane 2. In ideal circumstances (no drift, noplay, etc.) the set of projections 12-i are arranged along a straightline 10-3; in real circumstances the set of projections 12-i are notexactly arranged along that straight line 10-3, but more or lessscattered about this line. It is an object of the invention to identifysuch a straight line fitting the scattered projections 12-i.

FIGS. 2 a, 2 b and 2 c illustrate a sample to be studied by means of atilt series. The sample 2 consists of an object 14, and is provided withgold particles as markers. These markers may be identified by applying afirst (rough) selection method known per se. However, as a result ofthis first rough recognition step many candidate markers that are notreal markers may be identified as true markers (false positivecandidates). The first set of markers so obtained is represented in FIG.2 a by references 8-1, 8-2, . . . in general 8-i. After this firstrecognition step a following selection step is made; in this latter stepone or two probability parameters are attributed to each candidatemarker in each image, and a second set is selected as a subset ofcandidate markers from the first set of candidate markers on the basisof said probability parameters. The result of this selection step isrepresented in FIG. 2 b, in which some markers of the first setconsidered to be false positive candidates are indicated by dashes 18-i,and the markers of the first set that are selected as true markers (thesecond set of markers) are indicated by crosses 16-i. In FIG. 2 c theresult of the projection of the true markers 16-i onto a single imageplane 2 is illustrated. As already described with reference to FIG. 1 bmarkers 16-i rotate about the tilt axis 6. In principle each marker willproduce a rectilinear trace 20-i in said image plane, but some markersare positioned in such a way that their traces coincide, see, forexample, marker pairs 16-1, 16-2 and 16-3, 16-4. The straight linesobtained in this way are detected by means of an algorithm comprisingthe Hough transformation, as will be described with reference to FIGS. 3a and 3 b.

In FIG. 3 a a straight line 22-1 is represented in a usual system of(x,y) axes. A point of this line is denoted as x₀, y₀. A bundle ofstraight lines all going through point x₀, y₀ is denoted in general as22-i. The usual equation of such a straight line 22-1 is y=rx+c, inwhich r=tan(α), α being the angle between the line 22-1 and the positivex-axis, and c being the intersection of the line 22-1 with the y-axis.As is generally known, another representation of the line 22-1 isexpressed as r=y cos(α)−x sin(α) (r being the distance from the originx=0,y=0 to line 22-1), in which x and y represent points on the line22-1. In this way any line containing point x₀, y₀ is expressed as r=y₀cos(α)−x₀ sin(α), each set of values of the pair (r,α) now indicating aline of the bundle 22-i. The latter expression may be represented by onepoint in a set of (α,r) axes: see FIG. 3 b. So line 22-1, having oneparticular value of r and of α, is represented by one point 22-1 in FIG.3 b. In the same way all other lines going through point x₀, y₀ in FIG.3 a are represented by a point in FIG. 3 b, all lines 22-i in FIG. 3 aproviding points 22-i in FIG. 3 b, and all points 22-i togetherconstituting a sine-shaped line 24.

A bundle of lines all going through another point x₁, y₁, in FIG. 3 a(not drawn) will give rise to another sine-shaped line in FIG. 3 b; ifthese two sine curves intersect, it means that the two points x₀, y₀ andx₁, y₁ are situated on the same line in FIG. 3 a, which is always truefor two points. Finding points that are all situated on the same linemeans—in terms of the Hough representation of FIG. 3 b—that one shouldlook for intersection points of all sine curves corresponding to thosepoints. If these sine curves all have the same intersection point, itmeans that all points x_(i), y_(i) are situated on the same line in FIG.3 a.

Applying the above described algorithm to the projected marker points ofFIG. 2 c (which are not individually drawn but are represented by lines20-i) means that for each candidate marker point the correspondingsine-shaped curve should be established. So, as an example, if there are50 marker points identified as being true markers (the second set ofmarkers) and the tilt series consists of 141 images (tilt interval from−70° to +70° in steps of 1°) then there are 50×141=7050 points in FIG. 2c that give rise to a sine curve as in FIG. 3 b. To find lines in FIG. 2b containing a high density of marker points one should find allintersection points of all sine curves according to FIG. 3 b. This meansthat for n sine curves one should calculate ½n(n−1) intersection pointsto be represented in FIG. 3 b.

This calculation of all intersection points will provide the areashaving a high density of intersection points in FIG. 3 b; a high densityof intersection points in FIG. 3 b corresponds to lines containingrelatively many marker points according to FIG. 2 b.

For a numerical analysis of the density of intersection points in the(α,r) representation the following operation is carried out. Thecomplete interval in α is subdivided into (equal) small intervals Δαaround each value of α (that particular value of α being denoted as α₀).For each value of α₀ a histogram is formed, which means that thedistribution of the number of markers for all values of r is determined.The result of this operation is a collection of histograms equal innumber to the number of intervals into which α is subdivided. One suchhistogram is represented in FIG. 4. It should be noted that thedistribution of the numbers of markers over the various values of r isnot very divergent, which means that translating a small window (widthΔr) at a value of α₀ in the direction of r does not result in a largevariation in the numbers of markers that appear together in the window.A graphic representation of the latter operation is illustrated in FIG.5 a. FIG. 5 a shows the x,y system in which all (projections of) markersare placed. Finding the directions in which concentration lines ofmarkers are present involves defining a window having width Δr and somedirection α. Two windows 26 and 28 are drawn, window 26 having directionα₁ and window 28 having direction α₂. It can be seen from FIG. 5 a thatthe direction of concentration lines of markers coincides with directionα₁. When window 28 is translated across the x,y system, one observesthat there are no areas in that x,y system exhibiting concentrations ofmarkers, so the histogram formed by this operation will not show a highdegree of variation in the numbers of markers observed in that window,i.e. at the various values of r. The same holds for most other possibledirections of such a window. However, translating window 26 havingdirection α₁ results in the passage of the window over the concentrationlines having the same direction α₁, which will result in relatively manymarkers being detected in that window at the corresponding value of r,alternated by areas in which relatively few markers appear in thewindow. Such a distribution is illustrated in FIG. 6, to be discussedlater with reference to that Figure.

The above graphical operation for finding the concentration lines ofmarkers will now be described in a mathematical way suitable forapplication in a computer. For all values of α, a histogram r,n (i.e.the number of markers n versus the distance parameter r)

is formed. Each histogram is subjected to an entropy calculationaccording to the following expression:

$\begin{matrix}{S = {- {\sum\limits_{i = 1}^{i = N}\;{\left( \frac{n_{i}}{N} \right){\ln\left( \frac{n_{i}}{N} \right)}}}}} & (1)\end{matrix}$in which expression S is the entropy to be calculated, n_(i) is thenumber of markers detected at some value of the distance r_(i), N is thetotal number of markers in the histogram for which S is to becalculated, M is the number of intervals Δα and “ln” means the naturallogarithm, which is the logarithm to the base e, where e isapproximately equal to 2.718 . . . . The quantity S is, as is generallyknown, a measure of the divergence of the distribution of the numbers ofmarkers over the various values of r; in other words S is a measure ofthe “peakiness” of the histogram. Now all histograms are subjected to acalculation to obtain the value of S according to the above expression,and this range of values is represented as a function of the direction αin FIG. 5 b. It should be noted that a low value of S in this FIG. 5 bmeans that the histogram belonging to it has a high “peakiness”. In FIG.5 b the minimum value of S (corresponding to the maximum “peakiness”)occurs at α=77°, so that one may assume this to indicate the value ofthe direction perpendicular to the tilt axis with respect to the chosensystem of (x,y) axes.

FIG. 6 shows the histogram corresponding to the extremal value of Sreferred to above, i.e. the line direction perpendicular to the tiltaxis. Comparing the histogram according to FIG. 6 with the histogram ofFIG. 4 it is clear that the first histogram is much more “peaky”, andthus has a very high probability of being the one that corresponds tothe direction perpendicular to the tilt axis. From FIG. 6 one mayidentify the various parallel lines of markers as schematically depictedin FIG. 1 b, 2 c or 5 a. Once the various lines have been identified itis easy to identify the various positions of the projection of thesingle markers that gave rise to the relevant line (all other candidatemarkers being classified as false positive ones), and based thereon tomutually align the individual images of the tilt series. Such alignmentis known per se to the person skilled in the art.

1. A method for automatic alignment of tilt series in an electronmicroscope, comprising: applying markers to a sample to be imaged by theelectron microscope; providing a tilt series of images of the sample;identifying a first set of candidate markers in each of the images inthe tilt series; attributing at least one probability parameter to eachcandidate marker in each image; characterized in that the method furthercomprises: selecting a second set as a subset of candidate markers fromthe first set of candidate markers on the basis of said at least oneprobability parameter; projecting the candidate markers in the secondset onto a sole image; applying a fitting algorithm to determine a setof parallel straight lines or very elongate ellipses best fitting thecandidate markers in the sole image to identify a third subset ofcandidate markers; aligning the images in the tilt series on the basisof the third subset of identified candidate markers.
 2. A methodaccording to claim 1 in which the fitting algorithm used to determinethe set of parallel straight lines comprises the Hough transformation.3. A method according to claim 1 in which the fitting algorithm used todetermine the set of parallel straight lines or to determine a set ofvery elongate ellipses is constituted by the Generalized Houghtransformation.
 4. A method according to claim 1 in which, beforeidentifying candidate markers in each of the images in the tilt series,a cross correlation process is applied to the images of the tilt series.5. A method according to claim 1 in which the probability parameter isderived from at least one of the quantities: size of the marker andlocal contrast of the marker.
 6. A method for automatic alignment oftilt series in an electron microscope, comprising: applying markers to asample to be imaged by the electron microscope; providing a tilt seriesof images of the sample; identifying a first set of candidate markers ineach of the images in the tilt series; attributing at least oneprobability parameter to each candidate marker in each image;characterized in that the method further comprises: selecting a secondset as a subset of candidate markers from the first set of candidatemarkers on the basis of said at least one probability parameter;projecting the candidate markers in the second set onto a sole image;applying a fitting algorithm to determine a set of parallel straightlines or very elongate ellipses best fitting the candidate markers inthe sole image to identify a third subset of candidate markers, thefitting algorithm including: deriving for each candidate marker in thesecond set a sine-shaped curve based on coordinates of the correspondingcandidate marker, according to the Hough transformation; deriving fromthe sine-shaped curves a number of histograms indicating, for eachdirection, the relation between the density of candidate markers and theline distance parameter according to the Hough transformation; applyingan entropy operation to each of the histograms, resulting in a set ofentropy parameters, one entropy parameter for each histogram;establishing the minimum value in the set of entropy parameters;identifying the histogram corresponding to said minimum value as the oneshowing the highest degree of peak diversity; selecting from the latterhistogram a number of peaks; and deriving from each peak position in thehistogram the corresponding line distance parameter according to theHough transformation.
 7. A method according to claim 2 in which theprobability parameter is derived from at least one of the quantities:size of the marker and local contrast of the marker.
 8. A methodaccording to claim 3 in which the probability parameter is derived fromat least one of the quantifies: size of the marker and local contrast ofthe marker.
 9. A method according to claim 4 in which the probabilityparameter is derived from at least one of the quantities: size of themarker and local contrast of the marker.